Fix flaky/failing CoreOS TMT SSH provisioning in CI (bootc-dev/bootc)

Context

• CI workflow: .github/workflows/ci.yml contains a test-coreos job that builds an image and runs just test-tmt-on-coreos plan-bootc-install-on-coreos.
• A recurring failure is:
  • Waiting for SSH... attempt 60/60
  • SSH verification failed for plan /tmt/plans/tests-install/plan-bootc-install-on-coreos: Verifying SSH connectivity: SSH connectivity check failed after 60 attempts
  • This results in: error: Recipe test-tmt-on-coreos failed ... exit code 1
• The failure appears in workflow run 24793399494 (commit 2a459ac9b31eac38a294e3febec3811cbb3ccc6e, PR tests: Skip BLI detection test when discoverable-partitions config is absent #2155) and is not caused by the test change; it looks like libvirt/network/provisioning/guest boot regressions or missing diagnostics.

Goal

• Make the CI failure actionable and/or eliminate the root cause by ensuring libvirt networking is up before starting the CoreOS VM and by capturing strong diagnostics when SSH never comes up.

Required changes

1. Add robust debug capture when SSH connectivity verification fails in the CoreOS TMT harness.

   • Identify the script used by just test-tmt-on-coreos (likely hack/test-tmt-on-coreos or a similar wrapper).
   • Add an ERR trap or explicit failure handler that prints:
     • virsh list --all
     • virsh net-list --all
     • virsh net-dhcp-leases <network> (at least default)
     • virsh dominfo <domain> / virsh domiflist <domain> / virsh dumpxml <domain> for the relevant domain.
   • Ensure it still cleans up the VM after failure.

2. Ensure libvirt network is active before starting CoreOS tests.

   • In .github/workflows/ci.yml test-coreos job, add a step before running the CoreOS test to check and start the libvirt network (default or the one used by the harness).
   • Print the network list and DHCP leases.

3. Keep changes minimal and safe.

   • Do not weaken test coverage beyond what’s necessary.
   • Prefer making the environment deterministic and adding diagnostics.

4. Update documentation/comments as needed.

Deliverables

• Commit changes on a branch suitable for PR.
• Update/adjust any related Justfile targets if needed.

References

• Failing job log excerpt indicates failure in test-tmt-on-coreos recipe around line 211 and the failing TMT plan tmt/plans/tests-install/plan-bootc-install-on-coreos.
• CI workflow file: .github/workflows/ci.yml.

User context

• Current date: 2026-04-22
• User login: jmarrero

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: Please find a solution for this failing job. Use the logs, job definition, and any referenced files where the failure occurred. Keep your response focused on the solution and include code suggestions when appropriate.

Assistant: The job is failing because the Fedora CoreOS VM never becomes reachable over SSH during the TMT plan tests-install/plan-bootc-install-on-coreos:

• Log shows repeated: Waiting for SSH... attempt 60/60
• Then: SSH verification failed ... SSH connectivity check failed after 60 attempts
• This bubbles up as: error: Recipe test-tmt-on-coreos failed ... exit code 1

This is almost always one of:

1. the VM didn’t boot (bad disk/image, kernel args, ignition, etc.),
2. the VM booted but has no working network (no DHCP/forwarding on the libvirt network, wrong NIC model, etc.),
3. SSH isn’t available/allowed (sshd not running, wrong user/key, ignition didn’t apply).

Targeted solution: make the CoreOS provision step fail earlier with actionable diagnostics, and ensure the libvirt network is usable

Right now, the harness times out without capturing why the guest isn’t reachable. Update the just test-tmt-on-coreos ... wrapper (the logs indicate it’s implemented by hack/test-tmt-on-coreos, failing around “line 211”) to dump libvirt/QEMU state when SSH doesn’t come up.

Code suggestion (add a failure trap + libvirt diagnostics)

In hack/test-tmt-on-coreos, wrap the run so that on failure you automatically print:

• virsh list --all
• virsh dominfo <domain>
• virsh domiflist <domain>
• virsh net-list --all
• virsh net-dhcp-leases default (or the network you use)
• virsh console capture if possible (or virsh dumpxml <domain> at minimum)

Example pattern:

#!/usr/bin/env bash
set -euo pipefail

domain=""
dump_debug() {
  echo "=== libvirt debug ===" >&2
  virsh list --all || true
  virsh net-list --all || true
  if [[ -n "${domain}" ]]; then
    virsh dominfo "${domain}" || true
    virsh domiflist "${domain}" || true
    virsh dumpxml "${domain}" || true
  fi
  # If using default network, leases are critical for “no DHCP”
  virsh net-dhcp-leases default || true
}

trap dump_debug ERR

# ...existing code that sets/derives the libvirt domain name...
# domain="bootc-tmt-...."
# run tmt / bcvk provisioning that waits for ssh

This won’t “fix” networking by itself, but it turns the current opaque timeout into a...